Skip to content

[WIP] HADOOP-19785. mvn site fails in JDK17 #8182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
aajisaka wants to merge 1 commit into
base: trunk
Choose a base branch
from
Draft

[WIP] HADOOP-19785. mvn site fails in JDK17 #8182

aajisaka wants to merge 1 commit into from
+11 −4

Conversation

@aajisaka
Copy link
Member

@aajisaka aajisaka commented Jan 15, 2026 •
edited
Loading

Description of PR

Fix the following error while running mvn site with JDK17

JIRA: HADOOP-19785

Error:  /home/runner/work/hadoop/hadoop/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java:26: error: cannot find symbol
Error:  import com.codahale.metrics.JmxReporter;
Error:                             ^
Error:    symbol:   class JmxReporter
Error:    location: package com.codahale.metrics
Error:  /home/runner/work/hadoop/hadoop/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java:69: error: cannot find symbol
Error:    private JmxReporter jmxReporter;
Error:            ^
Error:    symbol:   class JmxReporter
Error:    location: class KMSWebApp
Error:  /home/runner/work/hadoop/hadoop/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java:36: error: cannot find symbol
Error:  import org.apache.hadoop.hdfs.MiniDFSCluster;
Error:                               ^
Error:    symbol:   class MiniDFSCluster
Error:    location: package org.apache.hadoop.hdfs
Error:  /home/runner/work/hadoop/hadoop/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java:41: error: cannot find symbol
Error:  import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
Error:                                               ^
Error:    symbol:   class DataNodeTestUtils
Error:    location: package org.apache.hadoop.hdfs.server.datanode
Error:  /home/runner/work/hadoop/hadoop/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java:42: error: cannot find symbol
Error:  import org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset;
Error:                                               ^
Error:    symbol:   class SimulatedFSDataset
Error:    location: package org.apache.hadoop.hdfs.server.datanode
Error: [ERROR] 5 errors

How was this patch tested?

Ran the following command and verified the above error has been fixed:

mvn clean install -DskipTests -DskipShade
mvn site

However, the Hadoop custom Doclet is failing due to the change in https://bugs.openjdk.org/browse/JDK-8253736

[ERROR] java.lang.ClassCastException: class jdk.proxy1.$Proxy0 cannot be cast to
 class jdk.javadoc.internal.tool.DocEnvImpl (jdk.proxy1.$Proxy0 is in module jdk.proxy1 of loader 'app'; jdk.javadoc.internal.tool.DocEnvImpl is in module jdk.javadoc of 
loader 'app')
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.WorkArounds.<init>(WorkArounds.java:104)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.BaseConfiguration.initConfiguration(BaseConfiguration.java:251)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.HtmlConfiguration.initConfiguration(HtmlConfiguration.java:220)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.run(AbstractDoclet.java:104)
[ERROR]     at jdk.javadoc/jdk.javadoc.doclet.StandardDoclet.run(StandardDoclet.java:103)
[ERROR]     at org.apache.hadoop.classification.tools.IncludePublicAnnotationsStandardDoclet.run(IncludePublicAnnotationsStandardDoclet.java:152)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Start.parseAndExecute(Start.java:556)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:393)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:342)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Main.execute(Main.java:63)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Main.main(Main.java:52)
[ERROR] 2 errors

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

@aajisaka
Copy link
Member Author

aajisaka commented Jan 15, 2026

Hi @zhtttylz, you've implemented new custom Doclet to support JDK17 in #8038, however, unfortunately mvn site fails in Doclet part. The detail is below:

  1. RootDocProcessor.process(env) creates a Proxy object that implements DocletEnvironment interface
  2. This proxy is passed to StandardDoclet.run(filtered)
  3. Inside the StandardDoclet, the javadoc internals (specifically WorkArounds.()) try to cast this proxy to the concrete implementation class jdk.javadoc.internal.tool.DocEnvImpl: https://github.com/jonathan-gibbons/jdk/blob/bc1e60c1bf91678ef18652a00aa2ce55b0446caa/src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/toolkit/WorkArounds.java#L112
  4. The cast fails because a proxy object cannot be cast to a concrete implementation class

It seems our custom doclet implementation is prohibited after https://bugs.openjdk.org/browse/JDK-8253736:

One particularly annoying wart is the cast on DocletEnvironment to DocEnvImpl, which effectively prevents using subtypes to carry additional info. It is not clear (even now) what the best way is to replace that logic.

Now I feel it's becoming really hard to maintain Hadoop's custom Doclets, and therefore I would like to drop the custom implementation. The primary change is we are going to build Hadoop JavaDoc with @LimitedPrivate, @Private or @Unstable classes, which are now excluded by our custom Doclets.

@slfan1989 @cnauroth @zhtttylz What do you think?

aajisaka
aajisaka commented Jan 15, 2026
View reviewed changes
pom.xml
Comment on lines +715 to +717
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${project.version}</version>
Copy link
Member Author

@aajisaka aajisaka Jan 15, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added because Hadoop Dynamometer production classes depend on HDFS test jar

@hadoop-yetus
Copy link

hadoop-yetus commented Jan 16, 2026

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 14s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 43m 2s trunk passed
+1 💚 compile 19m 2s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 compile 19m 26s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌ mvnsite 10m 10s /branch-mvnsite-root.txt root in trunk failed.
-1 ❌ javadoc 9m 7s /branch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt root in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ javadoc 8m 32s /branch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt root in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚 shadedclient 141m 26s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 34m 27s the patch passed
+1 💚 compile 18m 20s the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javac 18m 20s the patch passed
+1 💚 compile 19m 49s the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 javac 19m 49s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-1 ❌ mvnsite 8m 16s /patch-mvnsite-root.txt root in the patch failed.
-1 ❌ javadoc 7m 15s /patch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt root in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ javadoc 7m 23s /patch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt root in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚 shadedclient 57m 23s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 913m 20s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 51s The patch does not generate ASF License warnings.
1190m 50s
Reason Tests
Failed junit tests hadoop.yarn.service.TestYarnNativeServices
hadoop.yarn.server.resourcemanager.TestRMHA
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
hadoop.yarn.server.router.webapp.TestRouterWebServicesREST
hadoop.yarn.server.router.subcluster.fair.TestYarnFederationWithFairScheduler
hadoop.yarn.sls.appmaster.TestAMSimulator
hadoop.hdfs.tools.TestDFSAdmin
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
hadoop.mapreduce.v2.app.TestRuntimeEstimators
Subsystem Report/Notes
Docker ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/artifact/out/Dockerfile
GITHUB PR #8182
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname Linux c4c5b5d51744 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 2996b57
Default Java Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/testReport/
Max. process+thread count 3935 (vs. ulimit of 5500)
modules C: . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/console
versions git=2.25.1 maven=3.9.11
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989
Copy link
Contributor

slfan1989 commented Jan 19, 2026

Hi @zhtttylz, you've implemented new custom Doclet to support JDK17 in #8038, however, unfortunately mvn site fails in Doclet part. The detail is below:

  1. RootDocProcessor.process(env) creates a Proxy object that implements DocletEnvironment interface
  2. This proxy is passed to StandardDoclet.run(filtered)
  3. Inside the StandardDoclet, the javadoc internals (specifically WorkArounds.()) try to cast this proxy to the concrete implementation class jdk.javadoc.internal.tool.DocEnvImpl: https://github.com/jonathan-gibbons/jdk/blob/bc1e60c1bf91678ef18652a00aa2ce55b0446caa/src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/toolkit/WorkArounds.java#L112
  4. The cast fails because a proxy object cannot be cast to a concrete implementation class

It seems our custom doclet implementation is prohibited after https://bugs.openjdk.org/browse/JDK-8253736:

One particularly annoying wart is the cast on DocletEnvironment to DocEnvImpl, which effectively prevents using subtypes to carry additional info. It is not clear (even now) what the best way is to replace that logic.

Now I feel it's becoming really hard to maintain Hadoop's custom Doclets, and therefore I would like to drop the custom implementation. The primary change is we are going to build Hadoop JavaDoc with @LimitedPrivate, @Private or @Unstable classes, which are now excluded by our custom Doclets.

@slfan1989 @cnauroth @zhtttylz What do you think?

@aajisaka Thanks for the detailed analysis — after reading through it, I fully agree that the cost of maintaining this custom Doclet has become unreasonably high. With OpenJDK continuing to clean up internal APIs (the trend starting from JDK-8253736 is only getting stronger), future compatibility will only get worse, and the next LTS might break it completely.
I'm in favor of dropping the custom Doclet and switching to the standard doclet to generate complete JavaDocs (including all classes annotated with @Private / @unstable / @LimitedPrivate). The main reasons are:

  • The maintenance burden is too heavy and takes away energy from more valuable work;
  • The visibility of these annotated classes has very limited impact on most downstream users — they shouldn't be depending on @Private APIs anyway;
  • On the positive side, having the full picture can actually help developers/contributors who want to dig into the implementation details.

cc: @zhtttylz @cnauroth

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

build trunk

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@aajisaka @hadoop-yetus @slfan1989